Prepare affected-test selection rollout by Evangelink · Pull Request #10450 · microsoft/testfx

Amaury Levé (Evangelink) · 2026-08-05T10:30:54Z

Summary

Prepares testfx to adopt the experimental Microsoft.Testing.Platform affected-test workflow from dotnet/sdk#55574, using the composable filter-provider support already merged in testfx#10235.

The rollout remains intentionally disabled until a usable SDK and the public affected-test extension package with its local-filesystem storage schema are available. Existing test runs therefore keep their current command and behavior.

Prepared now

adds repository-specific test.affectedTests change and instrumentation scopes to global.json;
prepares trusted main Windows Debug for future map collection;
prepares PR Windows Debug for future affected-test selection;
transports the local map directory through Azure Pipelines Cache@2, without Blob credentials;
uses cache scopes that allow normal and fork PRs to read main's map but not overwrite it;
falls back to the full test suite on cache miss, incompatible map, selection failure, and non-PR runs;
avoids publishing partial selected-test coverage as the repository coverage report;
keeps both activation call sites at enableAffectedTests: false;
validates SDK compatibility, configuration shape, cache wiring, gate isolation, fallbacks, and rollback.

SDK status

SDK 11.0.100-rc.1.26406.108 contains --collect-test-map and --affected-tests, but it fails dotnet tool restore on clean agents due to the TransactionalAction regression fixed by dotnet/sdk#55595. The repository therefore retains stable SDK 11.0.100-rc.1.26402.102.

Activation validation requires an SDK newer than 11.0.100-rc.1.26406.108. The first candidate must pass dotnet tool restore on a clean agent before the rollout switches are enabled.

Storage and safety

The future extension storage path is $(Pipeline.Workspace)\affected-test-map. Cache keys include a manual compatibility version, OS, architecture, configuration, and unique build ID. Prefix restore selects the newest compatible main map.

Pipeline caches expire after seven days without activity, so cache miss is an expected state and always runs the unchanged full test command. The one-switch rollback remains setting enableAffectedTests to false; dormant storage configuration is permitted.

Remaining activation gates

A fixed SDK daily newer than 11.0.100-rc.1.26406.108.
The public affected-test extension package.
Its published local-filesystem storage schema.
A successful main collection that seeds a compatible cache.

After that, enable collect, validate the map, then enable run.

Validation

Debug repository build: 0 warnings, 0 errors.
Stable-disabled, broken-SDK activation, malformed-array, dormant-storage rollback, and enabled-without-storage scenarios pass.
global.json and modified YAML parse successfully.
PSScriptAnalyzer and git diff --check pass.
Final bug-focused review found no significant issues.

Copilot

Pull request overview

Prepares an intentionally disabled rollout of affected-test selection for Microsoft.Testing.Platform CI.

Changes:

Adds affected-test scopes and gated pipeline modes.
Adds rollout validation and safety checks.
Documents activation, storage, validation, and rollback.

Show a summary per file

File	Description
`global.json`	Defines affected-test scopes.
`azure-pipelines.yml`	Configures disabled collect/run call sites and validation.
`eng/pipelines/steps/test-windows-debug-coverage.yml`	Adds gated affected-test commands.
`eng/validate-affected-tests.ps1`	Validates rollout configuration.
`docs/affected-test-selection.md`	Documents rollout procedures.
`docs/README.md`	Links the new guide.

Review details

Files reviewed: 6/6 changed files
Comments generated: 3
Review effort level: Balanced

Use the public SDK build containing affected-test commands and prepare a credential-free Azure Pipelines Cache transport with safe full-test and coverage fallbacks. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

Keep the stable SDK pin until a daily containing the TransactionalAction fix is published, and enforce that corrected SDK only when rollout activation is enabled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

github-actions · 2026-08-07T00:26:34Z

🔍 Build Failure Analysis

Summary — Every build leg except one Windows Release leg failed identically at RestoreRepoTools (dotnet tool restore) with a crash inside the newly-pinned SDK (11.0.100-rc.1.26406.108).

Root cause: newly-pinned SDK `11.0.100-rc.1.26406.108` crashes on `dotnet tool restore`

This PR bumps global.json's tools.dotnet and sdk.version from 11.0.100-rc.1.26402.102 to 11.0.100-rc.1.26406.108 (needed for the new --collect-test-map/--affected-tests switches). With that SDK, arcade's RestoreRepoTools target invokes dotnet tool restore, which throws an unhandled System.NullReferenceException inside the SDK's own CLI startup path before it can do any work:

Unhandled exception: System.TypeInitializationException: The type initializer for 'Microsoft.DotNet.Cli.TransactionalAction' threw an exception.
 ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.DotNet.Cli.TransactionalAction.SetTransactionManagerField(String fieldName, Object value)
   at Microsoft.DotNet.Cli.TransactionalAction.DisableTransactionTimeoutUpperLimit()
   ...
   at Microsoft.DotNet.Cli.Commands.Tool.Restore.ToolRestoreCommand.Execute()

MSBuild then reports MSB3073: The command "... dotnet tool restore " exited with code 1 in Tools.proj, which fails Build.proj/Tools.proj restore on nearly every leg before compilation begins. This looks like a genuine regression in the pinned SDK build itself (Microsoft.DotNet.Cli.TransactionalAction static constructor), not something in this PR's other source changes — reverting the global.json SDK bump avoids it, or the SDK pin needs to move to a fixed 11.0.100-rc.1.* servicing build once one is available.

Affected legs — Linux Debug (1_0), Linux Release (2_0), Windows Release legs 3_1/3_2, all three Windows Debug legs (4_0/4_1/4_2), macOS Release (5_0), and macOS Debug (6_0) all hit this identical MSB3073/RestoreRepoTools failure — 8 of the 10 legs. Only Windows Release leg 3_0 reports zero errors, likely because a warm restore cache on that particular agent avoided invoking dotnet tool restore fresh; that does not indicate it would succeed on a clean machine.

Affected files / errors

global.json:L5 and global.json:L28 — SDK pin bumped to 11.0.100-rc.1.26406.108, which crashes on dotnet tool restore.
artifacts/toolset/11.0.0-beta.26405.4/Tools.proj:29 (RestoreRepoTools target, Exec task) — MSB3073: The command "...dotnet tool restore " exited with code 1 on Linux Debug, Linux Release, Windows Release (2/3 legs), Windows Debug (3/3 legs), macOS Release, macOS Debug.

Proposed fix

Either:

Pin to a later/fixed 11.0.100-rc.1.* SDK build once the Microsoft.DotNet.Cli.TransactionalAction NRE is patched upstream, or
Revert the SDK bump in global.json and gate the new --collect-test-map/--affected-tests prerequisites behind a separate, still-to-be-validated SDK update, keeping this PR's config-only changes on the currently working 11.0.100-rc.1.26402.102.

I have no network access to confirm whether a newer, non-broken 11.0.100-rc.1.* build is already published — recommend a maintainer check the dotnet/installer/dotnet/sdk servicing feed and re-pin, or re-run the build once a fixed SDK ships, rather than assuming this is a transient flake (the same NRE reproduced consistently across 8 of 10 legs).

Build overview (Linux Debug leg, representative)

Build: FAILED
Duration: 1.3s
MSBuild: 18.11.0-1.26406.108+6c7fa710f
Projects: 4  Errors: 2  Warnings: 1

Failed projects:
  ✗ Build.proj (0.9s)
  ✗ Tools.proj (0.8s)

All MSBuild errors (identical pattern across 8 legs)

Code	Project	File:Line	Message
`MSB3073`	`Tools.proj`	`Tools.proj:29` (`RestoreRepoTools`/`Exec`)	`The command "...dotnet(.exe) tool restore " exited with code 1`
—	`Build.proj`	—	`Build failed.` (cascades from `Tools.proj`)

Legs affected: 1_0 (Linux Debug), 2_0 (Linux Release), 3_1/3_2 (Windows Release, 2 of 3), 4_0/4_1/4_2 (Windows Debug, all 3), 5_0 (Darwin Release), 6_0 (Darwin Debug).
Leg with no error reported: 3_0 (Windows Release).

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit 0def8c9}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 83.1 AIC · ⌖ 2.07 AIC · ⊞ 11.4K · [◷]( · ◷)

github-actions

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 83.1 AIC · ⌖ 2.07 AIC · ⊞ 11.4K · ◷

Comments that could not be inline-anchored

global.json:5

🔧 MSB3073 — This SDK build (11.0.100-rc.1.26406.108) crashes with an unhandled NullReferenceException inside Microsoft.DotNet.Cli.TransactionalAction's static constructor whenever dotnet tool restore runs, failing RestoreRepoTools on 8 of 10 CI legs. Revert to the previously-working pin until a fixed 11.0.100-rc.1.* servicing build is confirmed available.

    &quot;dotnet&quot;: &quot;11.0.100-rc.1.26402.102&quot;,

global.json:28

🔧 MSB3073 — Same SDK pin; the sdk.version entry must match tools.dotnet above. Revert both together to restore a working dotnet tool restore.

    &quot;version&quot;: &quot;11.0.100-rc.1.26402.102&quot;,

Copilot

Review details

Suppressed comments (1)

global.json:57

The force-all scope covers only the root and direct test/ Directory.Build files. This repository also has src/Directory.Build.props, src/Platform/Directory.Build.props, src/Platform/Directory.Build.targets, and test/IntegrationTests/TestAssets/Directory.Build.targets; changes to these files alter project evaluation but will not match the configured force-all patterns. Include recursive src and test Directory.Build patterns so an old map cannot be used for these build-wide changes.

          "Directory.Build.*",

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Balanced

Keep exit code 2 from selected-test runs and force full selection when nested Directory.Build files change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot

Review details

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Balanced

Keep exit code 8 from affected-test runs so the full-suite fallback cannot mask the SDK's all-skipped safeguard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot

Review details

Files reviewed: 6/6 changed files
Comments generated: 1
Review effort level: Balanced

Require the global.json SDK pins to match and gate activation on the tools.dotnet version installed by CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot

Review details

Suppressed comments (1)

azure-pipelines.yml:722

The Windows job that reaches this run call checks out the repository at lines 415-416 without a fetchDepth, unlike DetectChanges. Affected-test selection derives the changed source set from Git, so a shallow Azure PR checkout can omit the merge parents and make --affected-tests unable to compute the PR diff, forcing the full-suite fallback on every run. Make this lane deterministic by using fetchDepth: 0 on that checkout (or explicitly fetching the target history) before enabling the switch.

          - template: /eng/pipelines/steps/test-windows-debug-coverage.yml
            parameters:
              # Enable only after the prerequisites in docs/affected-test-selection.md are satisfied.
              enableAffectedTests: false
              affectedTestsMode: run

Files reviewed: 6/6 changed files
Comments generated: 0 new
Review effort level: Balanced

Copilot AI balanced review requested due to automatic review settings August 5, 2026 10:30

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 5, 2026 10:31 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

Comment thread eng/pipelines/steps/test-windows-debug-coverage.yml Outdated

Comment thread eng/validate-affected-tests.ps1 Outdated

Comment thread eng/validate-affected-tests.ps1 Outdated

Copilot AI review requested due to automatic review settings August 6, 2026 16:58

Amaury Levé (Evangelink) force-pushed the dev/amauryleve/adopt-affected-tests branch from 02ca8e7 to 0def8c9 Compare August 6, 2026 16:58

Amaury Levé (Evangelink) marked this pull request as ready for review August 6, 2026 16:58

Copilot AI reviewed Aug 6, 2026

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 6, 2026 21:54 View session

Copilot AI review requested due to automatic review settings August 7, 2026 00:16

github-actions Bot reviewed Aug 7, 2026

View reviewed changes

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 7, 2026 00:28 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Comment thread eng/pipelines/steps/test-windows-debug-coverage.yml

Preserve affected test failures

95afe48

Keep exit code 2 from selected-test runs and force full selection when nested Directory.Build files change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot AI review requested due to automatic review settings August 7, 2026 08:42

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 7, 2026 08:43 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Comment thread eng/pipelines/steps/test-windows-debug-coverage.yml Outdated

Preserve all-skipped affected test failures

238ef10

Keep exit code 8 from affected-test runs so the full-suite fallback cannot mask the SDK's all-skipped safeguard. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot AI review requested due to automatic review settings August 7, 2026 09:12

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 7, 2026 09:12 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Comment thread eng/validate-affected-tests.ps1 Outdated

Amaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Aug 7, 2026

Validate the bootstrapped affected-test SDK

39d863d

Require the global.json SDK pins to match and gate activation on the tools.dotnet version installed by CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3baa2cd4-db32-4f9f-b6b7-8cc190f3799f

Copilot AI review requested due to automatic review settings August 7, 2026 10:42

Copilot started reviewing on behalf of Amaury Levé (Evangelink) August 7, 2026 10:43 View session

Copilot AI reviewed Aug 7, 2026

View reviewed changes

Petr Pokorny (0101) approved these changes Aug 7, 2026

View reviewed changes

Amaury Levé (Evangelink) merged commit 18ea171 into main Aug 7, 2026
34 checks passed

Amaury Levé (Evangelink) deleted the dev/amauryleve/adopt-affected-tests branch August 7, 2026 14:42

Navigation Menu

Uh oh!

Prepare affected-test selection rollout - #10450

Prepare affected-test selection rollout#10450
Amaury Levé (Evangelink) merged 5 commits into
mainfrom
dev/amauryleve/adopt-affected-tests

Amaury Levé (Evangelink) commented Aug 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

github-actions Bot commented Aug 7, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Amaury Levé (Evangelink) commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Prepared now

SDK status

Storage and safety

Remaining activation gates

Validation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 7, 2026

🔍 Build Failure Analysis

Root cause: newly-pinned SDK 11.0.100-rc.1.26406.108 crashes on dotnet tool restore

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Comments that could not be inline-anchored

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Review details

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Amaury Levé (Evangelink) commented Aug 5, 2026 •

edited

Loading

Root cause: newly-pinned SDK `11.0.100-rc.1.26406.108` crashes on `dotnet tool restore`